Visual surveillance: dynamic behavior analysis at multiple levels

New webcams and surveillance cameras are installed daily all around the world, adding huge quantities of data to the information stream that needs to be processed. As this happens, it is critical to develop methods that process such data-streams automatically and in real-time, reducing the manual effort that is still required for video content analysis. Of particular interest is to analyze the behavior of moving persons. Different aspects of a scene can be recorded at different scale levels. While high-resolution portrait pictures facilitate the analysis of facial details, they do not allow us to observe the motion and interaction between different persons. In contrast, a camera observing an entire scene may result in images where individual persons are too small to be detected. However, spatial structure and motion patterns can still be extracted. Thus, each scale level contains useful information but requires a different approach to exploit it. In this work, we present algorithms to analyze the behavior of dynamically moving persons in real-world scenes at different scale levels. The focus lies on developing methods that work online and fully automatically, adapting to the observations acquired so far. We present five novel methods. First, we present a real-time method to detect unfamiliar faces and to estimate their poses. This algorithm is applicable to a large pose range and is robust to facial variations and expressions. The second innovation consists of a method that detects and tracks moving persons in complex scenes. The algorithm is suitable for a wide variety of online applications and does not require scene-specific knowledge. Another branch of this thesis introduces a statistical model to analyze the activity in a scene, where agents interact according to complex dependency patterns. It allows us to automatically find periodic actions and relations between co-occurring and consecutive activities. Then we present a technique to detect novelty from webcam footage at the scene level. During run-time the method automatically learns what usually happens in a scene and continuously adapts to the scene content. Finally, we propose an approach to estimate the scene structure from videos. This estimation may be used to guide an object detection system to those image regions and local scales at which objects are more likely to occur.